Learning Visual Representations using Images with Captions

Current methods for learning visual categories work well when a large amount of labeled data is available, but can run into severe difficulties when the number of labeled examples is small. When labeled data is scarce it may be beneficial to use unlabeled data to learn an image representation that is low-dimensional, but nevertheless captures the information required to discriminate between image categories. This paper describes a method for learning representations from large quantities of unlabeled images which have associated captions; the aim is to learn a representation that aids learning in image classification problems. Experiments show that the method significantly outperforms a fully-supervised baseline model as well as a model that ignores the captions and learns a visual representation by performing PCA on the unlabeled images alone. Our current work concentrates on captions as the source of metadata, but more generally other types of meta-data could be used (e.g., video sequences with accompanying speech).